Assessing Model-Based Testing: Results from an Industrial Case Study

Companies continuously look for ways to improve their processes and cut costs using new technologies. Model-based testing (MBT) is a technology that has the potential to be such a cost saver. In this paper, we present a case study in which we compare manual testing processes with MBT processes based on finite state machine (FSM) models. The system under test (SUT) is a professionally developed webbased data collection system that is now in use, allowing researchers to exchange findings of laboratory analyses regarding food borne illnesses. Two versions of the SUT were tested to evaluate the regression testing abilities of the two approaches. Manual and model based testing were carried out by independent teams that had access to the same resources but did not interact with each other. We compare the effectiveness (issues found) and efficiency (effort spent) of the two approaches. The results show, for example, that the manual testing process observed in this study required less preparation time but its coverage was somewhat uneven, while the observed MBT study required more preparation time but was more systematic and detected more issues. Furthermore, the two testing approaches detected different types of issues. We also discuss lessons learned, how each technology can be improved, as well as how the best parts of each technology can be used in a hybrid approach to improve the testing processes.